1. Technical Field
This disclosure concerns a system and method for parallelizing applications by using a software library of operators designed to implement detail parallelized computation plans. In particular, this disclosure relates to an efficient and cost effective way to implement parallelized applications.
2. Background Information
Currently a large disparity exists between the amount of data organizations need to process at any given time and the computing power available to the organization using single CPU (uniprocessors) systems. Today, organizations use applications that process terabytes and even petabytes of data in order to derive valuable information and business insight. Unfortunately, many of the applications typically run sequentially on uniprocessor machines, and require hours and even days of computation time to produce useable results. The gap between the amount of data that organizations must process and the computational performance of uniprocessors available to the organizations continues to widen. The amount of data collected and processed by organizations continues to grow exponentially. Organizations must address enterprise database growth rates of roughly 125% year over year or equivalent to doubling in size every 10 months. The volume of data for other data rich industries also continue to grow exponentially. For example, Astronomy has a data doubling rate of every 12 months, every 9 months for Bio-Sequences, and every 6 months for Functional Genomics.
Although storage capacity continues to grow at an exponential rate, the speed of uniprocessors no longer grows exponentially. Accordingly, even though organizations may have the ability to continue to increase data storage capacity, computational performance of uniprocessor configurations can no longer keep pace. Organizations must identify a technical solution to address the diverging trends of storage capacity and uniprocessors performance.
In order to process large amounts of data, applications need large amounts of computing power and high I/O throughput. Programmers face the technical challenges of identifying efficient ways to partition computational processing and coordinate computing across multiple CPUs to address the growing gap between the demand and supply of computing power. Given the reality of limited network bandwidth availability, programmers also face the technical challenge of addressing the large bandwidth requirements needed to deliver vast amounts of data to multiple CPUs performing parallel processing computations. Merely introducing an additional machine to a processing pool (configuration) does not increase the overall network bandwidth of the configuration. Although, the local disk I/O bandwidth may increase as a result. A network topology maybe represented as a tree that has many branches that represent network segments and leaves that represent processors. Accordingly, a single bottleneck along any one network segment may determine the overall network capacity and bandwidth of a configuration. In order to scale bandwidth, efficient use of local disk I/O bandwidth increases must be leveraged.
The extraordinary technical challenges associated with parallelizing computational operations include parallel programming complexity, adequate development and testing tools, network bandwidth scalability limits, the diverging trends of storage capacity and uniprocessors performance, and efficient partitioning of computational processing and coordination in multiprocessor configurations.
A need has long existed for a system and method that economically, efficiently implements parallel computing solutions and effectively relieves the burden of developing complex parallel programs by programmers.